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1.  Introduction 

This  study  uses  controlled  experimentation  and  protocol  analysis  to  l)compare  linked  and 
non-linked  versions  of  the  NetBook  data  model  in  order  to  contribute  to  the  current  debate 
over  user  categorization  errors,  searching  via  links  and  getting  lost  in  nonlinear  systems,  and 
2)to  evaluate  the  effectiveness  of  the  NetBook  knowledge  exploration  system  [S85,S86]. 

The  NetBook  model  is  based  on  relational  databases  augmented  with  hierarchies  on 
domains  and  on  hypertext.  The  user  of  information  accesses  sets  of  text  fragments  through 
menu  selection  among  properties  and  values  and  then  explores  other  fragments  through 
hypertext  links.  The  system  attempts  to  linearize  the  set  of  fragments  chosen  by  a  query  in  a 
rational  way,  e.g.  prerequisite  information  first. 

1.1.  Motivations  for  Experimental  Study 

In  recent  years,  the  experimental  study  of  human  performance  in  using  computer  and 
information  systems  has  been  recognized  as  a  necessary  accompaniment  to  the  design  and 
development  of  these  systems.  Particular  emphasis  has  been  placed  upon  query  facility  usage 
and  interface  design,  both  in  evaluation  research  on  particular  systems  and  comparative 
research  between  systems  or  models  of  proposed  systems  [ShnSO].  Although  simulations 
have  been  used  (paper-and-pencil  tests,  for  example),  workers  such  as  Norman  believe  that  it 
is  essential  to  test  ideas  relevant  to  computer-human  interaction  within  an  actual  working 
environment,  rather  than  a  simulation  [No83|. 

Shneiderman  suggests  that  the  use  of  controlled  experimentation  is  particularly  appropri- 
ate in  the  database  and  information  systems  area,  since  the  potential  user's  background  is 
very  different  from  that  of  the  programmer  or  designer  [ShnSO].  Because  the  potential  user 
is  likely  to  be  £m  infrequent  and  unskilled  user  of  the  system,  an  interface  appropriate  for  a 
more  expert  user  might  be  useless  with  a  novice. 

Shneiderman  considers  the  more  anecdotal  protocol  analysis  to  be  another  appropriate 
experimental  tool.  Card,  Moran  and  Newell  use  protocol  analysis  to  model  the  computer 
user  within  an  information-processing  psychology  framework  [CMN83].  Their  application  is 
text  editors. 

Chapanis,  Anderson  and  Licklider  suggest  some  of  the  dependent  variables  which  can  be 
used  to  measure  effectiveness:  how  easy  it  is  to  learn  to  use  a  system,  the  extent  to  which 
tasks  can  be  performed  quickly,  and  the  extent  to  which  errors  are  likely  to  be  made 
[CAL83]. 

1.2  Data  Organization,  User  Interface  and  User  Errors 

The  present  study  was  concerned  with  two  related  aspects  of  information  systems  which 
affect  effectiveness  -  how  the  data  are  organized  in  the  database  and  the  ease-of-use  of  the 
interface. 

Canas  et  al.  studied  organization  in  a  manual  filing  cabinet  system  in  which  they  found  a 
retrieval  success  rate  of  over  90%  [CSC85].  The  authors  suggest  that  if  we  understand  the 
way  people  classify  and  search  for  information  in  manual  systems,  we  would  be  able  to 
design  better  automated  systems.  Studies  of  electronic  retrieval  systems  generally  show  a 
lower  rate  of  successful  retrieval  [FV80]. 

Retrieval  rates  may  be  lower  because  users  opt  to  use  only  basic  searching  features  of  an 


automated  system  in  which  they  are  unskilled.  Weyer  examined  the  effect  of  the  interface 
on  searching  strategies  using  a  dynamic  book.  Dynabook.  with  a  high-school  level  world  his- 
tory text  as  his  database  [We82].  He  compared  the  dynamic  version  to  two  others-  one  in 
paper  and  a  second  that  was  electronic  but  not  dynamic.  In  the  dynamic  book,  instead  of 
accessing  by  a  one-level  subject  index  and  page  numbers  as  in  the  non-  dynamic,  electronic 
book,  accessing  was  through  an  expanded  three-level  outline  of  chapters,  sections  and  subsec- 
tions, along  with  mechanisms  for  matching  patterns. 

Weyer  found  that  students  learned  elementary  features  of  the  system  quicklv.  but  the 
more  elaborate  features  were  seldom  used.  .Although  students  preferred  the  two  electronic 
forms  of  the  book,  the  non-dynamic  electronic  book  was  faster  to  use.  Answer  correctness 
did  not  vary  benveen  conditions. 

Many  accessible  databases  for  novice  users  are  menu  driven,  as  is  NetBook.  Otherwise, 
the  user  may  experience  what  Landauer  et  al.  call  "novice  freakout"  [LDGF82].  Indeed, 
"novice  freakout"  may  occur  even  in  a  menu-driven  system  if  the  menu  is  not  clear  to  the 
user,  or  if  the  data  in  the  database  are  poorly  arranged  from  the  user's  point  of  view.  As 
Walker  points  out,  it  is  essential  that  the  user  have  a  reasonable  amount  of  knowledge  about 
how  the  information  in  the  database  is  organized  [W8I].  Even  with  that  knowledge,  how- 
ever, if  the  organization  of  the  database  is  different  from  what  the  user  expects,  there  may 
still  be  problems. 

Landauer  et  al.  point  out  that  efficient  system  processing  generally  takes  priority  over 
other  consideration  in  arranging  data  in  a  database.  The  logical  connections  in  the  user's 
mind  may  not  match  the  connections  necessary  for  an  efficient  system  because  people  seldom 
organize  information  in  a  deeply  nested  hierarchy.  Also,  people  differ  in  their  categoriza- 
tions [DL84].  Studies  of  videotext  systems  support  the  contentions  of  Dumais  and  Landauer. 
In  an  experimental  study  of  the  Prestel  viewdata  service,  a  database  containing  information 
about  a  range  of  topics  -  news,  financial  data  and  other  reference  materials  -  Frankhuizen  and 
Vrins  asked  subjects  to  find  answers  to  various  questions,  such  as  "What  language  is  spoken 
in  Ethiopia?"  Almost  half  of  the  questions  were  not  answered  correctly  [FV80].  In  addition, 
in  almost  half  of  the  questions,  subjects  did  not  find  the  answer  using  their  initial  strategy, 
but  started  over  again  from  the  beginning  one  or  more  times.  In  terms  of  precision  of 
retrieval,  successful  locations  of  answers  were  not  efficient.  On  average,  twice  as  many 
pages  were  retrieved  as  was  necessary.  Frankhuizen  and  Vrins  attribute  these  problems  to 
category  overlap  and  ambiguities  in  category  naming.  One  videotext  system,  for  example, 
would  decompose  "night  life"  in  its  hierarchies  into  supposedly  disjoint  categories  including 
"attractions"  and  "entertainment".  The  problems  with  categorization  in  a  single  hierarchy 
suggest  that  a  knowledge  exploration  system  should  support  multiple  overlapping  hierarchies, 
a  decision  taken  in  NetBook. 

An  additional  categorization  problem  that  any  system  will  have  is  suggested  by  Walker 
[W81].  In  an  information  system,  new  developments  in  a  body  of  knowledge  might  necessi- 
tate adding  new  entries  to  the  index  listing,  but  it  will  be  difficult  to  reclassify  the  older  text 
to  reflect  the  changes.  The  example  that  Dumais  and  Landauer  use  is  that  if  you  want  to 
locate  computer  science  books,  they  are  classified  under  the  category  "generalities"  in  the 
Dewey  Decimal  System. 

1.3.  Inforination  Links 

One  proposal  for  overcoming  categorization  problems  and  false  starts  has  been  to  use 
links  between  information  fragments.  In  Bush's  classic  article  on  the  future  of  information 
systems,  he  suggested  that  because  the  index  categories  by  which  information  can  be  stored  in 


a  system  and  the  mental  processes  by  which  a  human  classifies  information  are  very  dif- 
ferent, selection  by  indexing  is  not  optimal  [Bu451.  Rather,  since  the  human  mind  operates 
in  an  essentially  nonlinear  way  by  association,  selection  by  association  is  a  better  strategy,  if 
it  can  be  mechanized.  Bush  called  this  process  of  linking  many  items  of  information  "build- 
ing a  trail",  ideally  with  the  capacity  to  branch  onto  side  trails  of  subtopics,  through  the  avail- 
able texts. 

These  ideas  are  consistent  with  theories  in  modern  psychology.  Take,  for  example,  the 
theory  of  "spreading  activation".  The  basic  idea  is  that  each  person  has  a  mental  dictionary 
with  words  linked  to  other  words  by  associated  meanings,  associated  sounds,  or  associated 
grammars,  forming  a  sort  of  spider  web  of  points.  Speaking  is  an  activation  of  points  on  this 
web.    Each  word  selected  activates  an  alternate  path  through  the  web  [M87]. 

Yankelovich  and  Meyrowitz  say  that  computers,  compared  to  books,  are  particularly 
suited  to  helping  users  create  connectivity,  i.e.  webs  of  related  information  [YM85].  In  a 
book,  links  are  usually  created  only  through  footnotes  or  through  notes  that  the  user  writes  in 
the  margins.    With  electronic  media,  organization  of  materials  can  be  more  flexible. 

Links  between  fragments  of  information  may  also  have  their  drawbacks.  Yankelovich 
and  Meyrowitz  believe  that  a  disadvantage  to  electronic  information  systems  is  that  they  do 
not  inform  users  where  they  are  in  a  document.  Following  a  trail  of  links  may  exacerbate 
this  problem  [N79,YM85].  This  is  in  contrast  to  a  book,  in  which  the  reader  can  tell  if  he  is 
halfway  through  or  someplace  else.  Because  an  electronic  book  is  nonlinear,  what  is  the  mid- 
dle for  one  user  might  be  the  beginning  for  another  user.  Therefore,  someone  following  link 
after  link  can  feel  very  disoriented.  For  example,  in  the  PRESS  hypertext  document  system 
at  Brown  University,  users  often  found  it  difficult  to  remember  where  they  were  in  the  text. 
Nevertheless,  virtually  all  researchers  feel  strongly  that  any  electronic  document  system 
should  include  tools  for  connectivity. 

Several  nonlinear  text  systems  which  make  use  of  links  have  been  described,  the  largest 
being  ZOG  with  20,000  frames  [A.M84,E84,F84,H80,L85,Ne84,S85,S86,SH87,TW86]. 
Many  different  design  features  have  been  employed.  In  Fogg's  The  Living  in  a  Database 
System,  for  example,  users  from  inside  an  entity-relationship  tuple  can  answer  queries  by 
examining  the  database  from  that  tuple's  perspectives.  Fogg  bases  his  interface  on  the  ideas 
of  Cattell  who  describes  an  interface  in  which  each  frame  contains  a  list  of  adjacent  nodes  for 
the  user  to  explore  [C80].  Herot's  spatial  data  management  system  (SD.MS)  presents  infor- 
mation graphically. 

Shneiderman  describes  a  nonlinear  system  that  works  via  embedded  menus  -  The  Interac- 
tive Encyclopedia  System  (TIES).  Users  can  select  topics  through  words  that  are  highlighted 
in  the  text  and  immediately  go  to  view  an  article  on  the  selected  topic.  This  is  in  contrast  to 
the  more  traditional  separation  of  text  reading  from  menu  category  selection.  When  embed- 
ded menus  are  compared  to  more  traditional  menus,  subjects  complete  more  questions  more 
quickly  with  embedded  menus  and  also  jump  more  frequently  among  articles. 

2.  Our  Experiment 

A  database  consisting  of  text  fragments  was  created,  using  the  authoring  and  editing  facil- 
ities of  NetBook  [S85,S86|.  The  text  fragments  were  taken  largely  from  two  chapters  of  an 
introductory  sociology  text.  Some  additions  and  modifications  were  made  where  it  was 
thought  that  information  needed  to  be  updated  or  elaborated  upon. 

Fragments    were    broken    down    by    ten    relevant    properties    and    appropriate    values. 


(Properties  correspond  to  slots  in  frame-based  systems  or  attributes  in  database  systems.) 
Subvalues,  or  hierarchies  within  property  domains,  were  not  employed,  because  we  wanted 
to  use  only  the  simplest  mechanisms  in  NetBook. 

Properties  and  values  were  chosen  in  such  a  way  as  to  make  it  possible  to  obtain  the  same 
information  through  more  than  one  query.  Properties  and  values  were  also  chosen  to  be 
orthogonal  as  much  as  possible,  i.e.  choosing  a  value  of  one  property  should  not  restrict  the 
possible  values  of  other  properties. 

The  links  between  fragments  included  example,  critique,  counter-argument,  compared- 
with,  and  continuation  (linking  a  fragment  to  the  following  one  in  the  original  text).  Con- 
tinuation links  were  also  used  by  NetBook  as  input  to  a  presentation  algorithm  which 
presented  adjacent  fragments  with  the  properties  and  values  queried  in  linear  order.  The 
continuation  order  was  not  violated,  unless  it  was  necessary  to  avoid  violating  an  example 
link  or  one  of  the  other  types  of  links.  The  notion  of  trying  to  present  the  fragments 
returned  by  a  query  in  the  same  order  as  they  appear  in  the  original  text  may  strike  nonlinear 
text  adherents  as  blasphemy,  but  that  heuristic  seemed  reasonable  to  us,  all  else  being  equal. 

Subjects  were  undergraduate  volunteers  from  The  New  York  Institute  of  Technology. 
Fifty  vere  randomly  assigned  to  one  of  two  groups  and  run  individually.  Subjects  in  both 
groups  were  permitted  to  use  properties  and  values  to  specify  text  fragments.  The  difference 
came  about  in  the  use  of  hypertext  links. 

l)Links  group.    Participants  were  told  about  the  links  option  and  encouraged  to  use  it. 
2)No-Links  group.    Participants  were  tola  that  the  links  option  on  their  screen  menu  could 
not  be  used. 

An  additional  ten  subjects  were  randomly  assigned  to  a  third  group,  for  the  purpose  of 
making  baseline  comparisons  of  speed  and  accuracy.  This  group  used  a  paper  copy  of  the 
text  and  its  Table  of  Contents  to  search  for  information  (Paper  Text  group).  These  subjects 
were  also  run  individually.  The  groups  had  no  significant  differences  in  mean  grade-point 
average  and  had  an  equal  amount  of  exposure  to  sociology. 

Each  subject  from  group  1  and  group  2  was  exposed  to  a  20  minute  group  introduction  to 
the  database  and  the  query  system,  followed  by  an  individual  review  and  demonstration.  In 
the  individual  review,  subjects  were  given  a  sheet  listing  the  properties  and  values  used  to 
organize  text  fragments.  The  sheet  also  contained  a  brief  outline  of  how  to  use  the  menu 
system  to  form  queries.  Subjects  were  given  five  minutes  to  review  this  sheet  and  told  that 
they  could  also  keep  it  in  front  of  them  to  refer  to  as  the  session  proceeded.  They  were  also 
encouraged  to  ask  questions  about  the  procedure  at  this  time. 

After  each  subject  in  group  1  and  group  2  (the  electronic  groups)  reviewed  the  database 
organization  and  query  procedure  outline,  the  subject  underwent  a  hands-on  demonstration. 

Subjects  were  then  told  to  locate,  as  fast  as  they  could,  the  information  requested  in  the 
five  questions  given  them.  The  presentation  order  of  the  questions  was  rotated  in  order  to 
measure  practice  effects. 

Each  search  was  timed  from  the  point  at  which  the  question  was  presented  to  the  point  at 
which  the  subject  began  to  record  the  answer  on  the  answer  sheet.  In  addition,  protocols 
were  collected  of  each  subject's  menu  choices  throughout  each  query  (see  Figures  1  and  2  for 
a  sample  protocol  sheet  and  NetBook's  menu  choices).  If  the  subject  was  unable  to  complete 
all  five  queries  within  50  minutes,  the  session  was  terminated  at  that  point. 


Figure  1.  Sample  Protocol  Sheet 


Name: BM_ 


Group: No_Links 

Number  of  1st  Question  in  this  Permutation: 1 

Time  Queries  made: 

Q.  1)    17'50" 

start  query 
start  query 
specify  or  change  properties 

add  spec-theorist. Marx 

add  spec-class  category. gen.  comments 

display  text 

remove  spec-class  cat. gen.  comments 

display  text 


2)     6'05" 


3)     2'49' 


4)     6'18' 


start  query 

specify  or  change  properties 
add  spec-location. India 
add  spec-type  of  strat. caste 
display  text 


start  query 

specify  or  change  properties 

add  spec-location. Japan 

add  spec-social  mobility. intergen'l 

display  text 


start  query 

specify  or  change  properties 

add  spec-measurement  gen.  comments 

add  spec-location-U.S. 

remove  spec-measurement. gen.  comments 

add  spec-empirical  study. Warner  &  Lunt 

add  spec-type  of  strat. class 

display  text 

display  spec 

remove  spec-type  of  strat. class 

display  text 


5)     6'36" 


start  query 

specify  or  change  properties 

add  spec-location. U.S. 

add  spec-class  category. upper 

display  text 


3.  Results 

3.1.  Use  of  Links 

Links-group  subjects  were  not  all  equally  inclined  to  use  them.  Figure  3  shows  the 
number  of  subjects  who  used  the  hypertext  capabilities  of  the  system,  broken  down  by  ques- 
tion. 

Some  questions  apparently  encouraged  the  use  of  links  more  than  others  did.  Questions 
#3  and  #4  were  fairly  accessible  without  the  use  of  links,  so  most  subjects  searched  for  an 
answer  through  properties  and  values.  Informal  questioning  of  subjects  who  did  not  use 
links,  or  who  used  them  very  sparingly  revealed  that  some  subjects  felt  overwhelmed  by  the 
hypertext  menu,  despite  a  review  of  the  menu  choices  as  part  of  the  initial  introduction  of 
subjects  to  the  system. 

Figure  4  shows  the  frequencies  of  hypertext  use  in  queries  in  the  Links  group.    Three 


Figure  2.  NetBook  User  Menus  (version  without  editing  facilities) 

Main  Menu 

1. start  new  query 

2. specify  or  change  properties  of  a  query 

3. list  current  queries 

4. display  text  associated  with  a  query 

5. quit 

Specify  or  Change  Properties  Menu 

Ladd  specification 

2. list  possible  specifications 

3. remove  specification 

4. display  current  specifications 

5. display  text  associated  with  a  query 

6. return  to  previous  menu 

Hypertext  Menu 

1. return  to  reading  the  next  fragment  specified  by  the  query 

2. mark  place 

3. return  to  last  mark 

4. go  back  1  step 

5. repeat  text 

6. save  for  printing 

7. create  a  new  query  with  this  fragment's  properties 

8. [menu  continues  here  with  a  list  of  link  options] 


subjects  would  not  use  the  hypertext  system  at  all.  Other  subjects  used  it  for  between  one 
and  three  queries.  Subjects  who  did  not  use  links,  although  they  were  available,  were 
grouped  with  those  in  the  No-Links  group  for  analyses  where  applicable. 

NetBook's  links  were  sufficient  to  locate  information  fairly  directly  with  an  inadequately 
specified  or  incompletely  specified  query.  For  example,  in  the  following  protocol  excerpt, 
for  Question  #5  ("The  percent  of  the  national  wealth  that  is  owned  by  the  richest  fifth  of 
Americans."),  the  subject  has  specified  only  that  she  wishes  to  see  all  text  related  to  wealth: 

start  query 

specify  or  change  properties 

add  spec. -form  of  class  inequality. wealth 

display  text  -  hypertext  -  compared  with 

Althoug.n  it  took  the  subject  788  seconds  to  complete  the  query,  somewhat  more  than 
average  for  subjects  in  the  No-Links  group,  she  needed  to  follow  only  one  link  to  obtain  the 
answer.  .Most  of  the  time  the  subject  spent  completing  the  query  was  spent  hesitating  about 
what  to  do,  rather  than  doing  anything  with  the  system  that  took  a  substantial  amount  of 
time. 

On  the  other  hand,  a  subject  with  a  query  that  was  not  fully  specified,  or  inappropriate, 
could  also  become  totally  lost  in  the  hypertext  feature  of  NetBook.  In  the  following  protocol 
excerpt,  the  subject  spent  the  entire  fifty  minutes  attempting  to  locate  the  answer  to  Question 
#1  ("The  name  of  Marx's  book,  in  which  he  begins  to  present  his  ideas  on  social  class."). 
She  never  completed  the  query,  but  rather  went  back  and  forth  over  the  same  text  links: 

start  query 

specify  or  change  properties 
add  spec. -theorist. Marx 
display  text  -  hypertext  -  continuation  -  go 

back  one  step  -  go  back  one  step 
list  possible  spec. -properties 
add  spec. -class  category. general  comments 
add  spec. -theorist. Marx 
list  possible  specs. -properties 
add  spec. -theory. conflict 
display  text  -  hypertext  -  continuation  - 

continuation  -  go  back  one  step  -  create  a 

new  query  -  exit 
display  text  -  hypertext  -  counter-argument  - 

counter-argument  -  critique  -  continuation  - 

critique  -  create  a  new  query  -  mark  place  - 

exit 
start  query 

specify  or  change  properties 
list  possible  specs. -properties 
add  spec. -location. Continental  Europe 
add  spec. theorist. Marx 
display  text 

display  current  specifications 
list  possible  specs. -return 
add  spec. -type  of  stratification. class 
display  text  -  hypertext  -  continuation  - 


continuation  -  continuation  -  continuation 
-  mark  place  -  create  a  new  query  -  see 
text  of  which  this  is  critique  -  see 
text  of  which  this  is  critique  ■ 
continuation  -  continuation  -  go  back  one 
step  -  go  back  one  step  -  example  -  go 
back  one  step  -  exit 
add  spec. -theory. conflict 
(subject  used  up  all  her  time  at  this  point) 

The  protocol  excerpt  of  the  following  subject  illustrates  the  way  a  subject  may  use  links  to 
locate  information  with  a  poorly  specified  query.  Completion  of  this  query  for  question  #4 
took  737  seconds.  Instead  of  using  the  property  'empirical  study'  with  the  value  "Warner 
and  Lunt"  and  the  property  "class  category"  with  the  value  "general  comments"  in  the  query, 
the  subject  chose  to  approach  her  task  by  guessing  at  a  correct  measurement  method.  She 
begins  by  choosing  an  incorrect  method,  removes  it,  chooses  another  incorrect  specification, 
removes  it,  adds  one  that  provides  only  general  comments  and  from  there  follows  links  until 
finding  the  correct  answer; 

start  query 

specify  or  change  properties 

list  possible  specs. return 

display  text 

add  spec. -measurement  method. subjective 

display  text 

remove  spec. -measurement  method. subjective 

Figure  3.  Use  of  Links  by  Question  Number 


Question 

# 

U 

sed  Hyp 

ertext 

Did  Not  Use  Hypertext 

1 

15 

3 

2 

10 

6 

3 

3 

13 

4 

5 

15 

5 

10 

9 

Figure  4.  Frequency  of  Hypertext  Use 


No.  of  Queries  Using  Hypertext 

Frequency 

0 

3 

1 

7 

2 

9 

3 

6 

(average  no.  of  completed  queries  =  3.28) 


add  spec. -measurement  method. objective 
list  possible  specs. -values. location 
display  te.xt 

remove  spec. -measurement  method. objective 
add  spec. -measurement  meth. general  comments 
list  possible  spec. -values. measurement  meth. 
display  current  specifications 
display  text  -  hypertext  -  continuation 

This  can  be  compared  to  the  following  protocol  excerpt  for  question  #1,  in  which  the  sub- 
ject uses  links  not  to  locate  the  correct  answer,  but  to  realize  that  he  has  specified  his  query 
incorrectly.  He  then  returns  to  the  menu  to  adjust  his  specification  and  fiiids  the  answer. 
This  query  took  533  seconds  to  complete: 

start  query 

specify  or  change  properties 

add  spec  -theorist. Marx 

add  spec. -location. Soviet  Union 

display  text  -  hypertext  -  continuation  -  exit 

remove  spec. -location. Soviet  Union 

add  spec. -type  of  stratification. class 

display  text 

3.2.  Number  of  Links 

Figure  5  shows  the  average  number  of  links  used  for  each  question  by  those  subjects  in 
the  Links  group  that  used  the  system's  hypertext  capabilities.  The  maximum  number  of  links 
followed  on  any  query  was  28. 

There  was  a  statistically  significant  negative  correlation  between  the  average  number  of 
links  a  subject  followed  and  the  total  number  of  queries  completed  (r  =  -.74,  p  =  0.001). 
Unsurprisingly,  there  was  also  a  statistically  significant  negative  correlation  between  the  aver- 
age number  of  links  a  subject  followed  and  the  total  number  of  correct  queries  (r  =  -.73,  p 
=  0.001).  The  number  of  links  followed  slowed  down  query  completion  time  for  subjects  in 
the  Links  group. 

3.3.  Completion  Time 

Comparisons  were  also  made  of  the  average  time  it  took  subjects  to  complete  each  query. 
Figure  5.  Average  Number  of  Links  Used  by  Question 


Question 

# 

Mean 

Star 

dard  Deviation 

1 

3.933 

7.235 

2 

4.500 

5.276 

3 

3.000 

3.464 

4 

4.800 

4.324 

5 

3.100 

3.143 
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In  four  out  of  five  questions,  subjects  in  the  Links  group  took  the  longest  amount  of  time  to 
complete  the  query,  followed  by  subjects  in  the  N'o-Links  group,  followed  by  subjects  in  the 
Text  group.  In  one  of  the  questions  (Question  #3  -  "The  intergenerational  mobility  rate  in 
Japan."),  the  Links  group  took  longer  on  average  than  the  No-Links  group,  but  the  Text 
group  took  longest  of  all.  Since  the  information  requested  was  not  readily  obtainable  from 
the  Table  of  Contents  in  the  paper  text  version,  but  required  more  searching  of  related  topics 
in  the  paper  text,  this  was  understandable. 

Figure  6  lists  the  average  number  of  queries  completed  by  each  group  and  Figure  7  lists 
the  means  and  standard  deviations  for  completion  time  for  each  group  for  each  question. 
One-way  analyses  of  variance  revealed  statistically  significant  differences  among  groups.  T- 
tests  on  the  two  electronic  groups  revealed  significant  differences  for  three  of  the  five  ques- 
tions. 

3.4  Specifications 

Although  the  larger  average  number  of  specifications  by  subjects  in  the  Links  group  was 
not  statistically  significant,  over  both  groups  there  was  a  significant  positive  correlation 
between  the  average  number  of  specifications  used  for  a  query  and  the  average  number  of 
seconds  spent  obtaining  the  answer  to  a  query.  Perhaps  with  a  faster  system,  this  relation- 
ship would  not  hold,  but  in  this  particular  version  of  NetBook,  there  is  significant  overhead 
to  forming  highly  specific  queries.  The  correlation  between  average  time  per  query  and  aver- 
age number  of  specifications  used  is  .84  (p  =  0.001). 

As  an  illustration  of  this  overhead,  contrast  the  following  two  protocol  excerpts  for  ques- 
tion #5  from  different  subjects  in  the  No-_inks  group: 

start  query 

specify  or  change  properties 

add  spec. -location. United  States 

add  spec. -class  category. upper 

display  text 
(time  =  396  seconds) 

and 

start  query 
start  query 
specify  or  change  properties 

add  spec. -form  of  class  inequality  wealth 

Figure  6.  Average  Number  of  Queries  Completed  by  Group 


Group 

Mean 

Standard  Deviation 

Text 

5.00 

0.0 

No-Links 

4.25 

1.29 

Links 

3.23 

1.48 

(F  =  7.784,  n  =  60,  p  =  0.0010) 
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Figure  7.  Average  Seconds  to  Complete  Each  Query 


Question  # 

Group 

Mean 

Standard  Deviation 

1 

Text 

508.80 

493.56 

No-Links 

570.55 

526.05 

Links 

1125.56 

837.28 

(F  =  4.306,  n  =  48,  p  =  0.0194) 


Question  # 

Group 

.Mean 

Standard  Deviation 

2 

Text 

47.70 

15.10 

No-Links 

397.92 

289.37 

Links 

877.14 

512.36 

(F  =  17.810,  n  =  48,  p  =  0.0000) 


Question  # 

Group 

.Mean 

Standard  Deviation 

3 

Text 

726.00 

434.39 

No-Links 

308.08 

171.28 

Links 

493.00 

560.15 

(F  =  5.141,  n  =  50,  p  =  0.0096) 


Question  # 

Group 

.Mean 

Standard  Deviation 

4 

Text 

109.10 

18.47 

No-Links 

581.70 

502.98 

Links 

703.94 

515.19 

(F  =  5.553,  n  =  54,  p  =  0.0066) 


Question  # 

Group 

Mean 

Standard  Deviation 

5 

Text 

52.40 

7.03 

No-Links 

616.00 

667.49 

Links 

803.59 

413.66 

(F  =  6.675, n  =  52.  p  =  0.0027) 

display  text 
(time  =  299  seconds) 

It  is  clear  from  the  above  two  excerpts  from  subject  protocols  that  both  subjects  under- 
stood the  data  retrieval  system.  Nevertheless,  neither  subject  used  a  full  specification  to  find 
the  answer  to  the  question  ("The  percent  of  the  national  wealth  that  is  owned  by  the  richest 
fifth  of  Americans.")  The  first  subject  narrowed  down  the  range  of  material  to  be  searched 
more,  but  spent  so  much  more  time  doing  so,  that  retrieving  the  answer  actuallv  took  more 
time  -  roughly  one-third  more. 

Some  subjects  continued  to  make  specifications  when  no  longer  necessary,  or  made 
incorrect  specifications  ("false  starts")  and  then  had  to  begin  again.  The  following  protocol 
excerpt  illustrates  the  use  of  inappropriate  continued  specification.  Because  the  user  became 
impatient  each  time  she  asked  to  see  text,  she  narrowed  down  the  field  of  information 
requested  too  severely.    To  complete  this  query  for  question  #1  took  1770  seconds: 
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start  query 

specify  or  change  properties 

add  spec. -theorist. Marx 

display  text 

add  spec. -class  cat. general  cominents 

add  spec. -theorist.. Marx 

display  text 

add  spec. -type  of  stratification. class 

display  text 

add  spec. -theory. conflict 

display  text 

add  spec. -location. Great  Britain 

display  text 

display  current  specifications 

remove  spec. -theory. conflict 

display  current  specifications 

remove  spec-  location. Great  Britain 

display  current  specifications 

add  spec. -degree  of  indust. industrial 

display  text 

remove  spec. -degree  of  indust. industrial 

display  current  specifications 

add  spec. -theorist. Marx 
start  query 
specify  or  change  properties 

add  spec. -theorist. .Marx 

display  text 

add  spec,  theory. conflict 

display  text 

In  the  above  excerpt,  the  subject  decides  to  begin  over  again  with  a  considerably  scaled 
down  query.  Her  original  specifications  were  all  not  entirely  inappropriate  and  two  are  re- 
used. 

3.5.  Practice  Effects 

As  mentioned  above,  the  order  in  which  questions  were  presented  to  subjects  was  rotated 
in  all  groups  in  order  to  measure  practice  effects.  Figure  8  shows  mean  query  completion 
times  for  all  groups  combined  for  first  through  fifth  queries.  Subjects  who  began  a  query, 
but  did  not  complete  it  because  they  ran  out  of  time  are  included.  There  appears  to  be  a 
trend  towards  faster  completion  of  later  queries. 

However,  since  the  Text  group  completed  all  queries,  while  only  a  portion  of  both  elec- 
tronic groups  did  so,  the  faster  responses  of  the  Text  group  are  overrepresented  in  the  latter 
queries.  Therefore,  mean  completion  times  for  first  through  fifth  queries  were  broken  down 
by  group  and  query  for  those  subjects  who  had  completed  all  five  queries.  The  mean  query 
completion  times  are  shown  in  Figure  9.  They  show  practice  effects  for  both  No-Links  and 
Links  subjects  through  the  fourth  query  completed.  Completion  times  increase  for  the  fifth 
query,  perhaps  as  a  result  of  fatigue. 

3.6.  Gender,  Attitudinal  and  Experiential  Measures 
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Figure  8.  Mean  Query  Completion  Times  by  Query  Order 


Query  O 

'der 

N 

umber 

Mean 

Stand 

ard  Deviation 

First 

60 

827.57 

688.99 

Second 

57 

514.05 

483.10 

Third 

53 

518.43 

477.49 

Fourth 

44 

391.18 

438.93 

Fifth 

38 

402.97 

355.73 

Figure  9.  Mean  Completion  Times  in  Seconds  by  Group  and 
Query  order  for  Subjects  Completing  5  Queries 


Query  Number 
1 


Group  Mean  Standard  Deviation 

Text  408.90  576.28 

No-Links  466.43  234.31 

Links  847.38  282.01 


(F  =  3.773,  n  =  36,  p  =  0.0335) 


Query  Number 

Group 

.Mean 

Standard  Deviation 

2 

Text 

206.70 

265.54 

No-Links 

389.83 

166.09 

Links 

543.88 

350.13 

(F  =  4.334,  n  =  36,  p  =  0.0213) 


Query  Number 
3 


Group  Mean  Standard  Deviation 

Text  257.90  304.03 

No-Links  335.89  195.34 

Links  371.38  199.67 


(F  =  0.601,  n  =  36,  p  =  0.5543) 


Query  Number 
4 


Group  .Mean  Standard  Deviation 

Text  376.90  547.20 

No-Links  280.33  88.79 

Links  300.25  152.93 


(F  =  0.338,  n  =  36,  p  =  0.7158) 


Query  Number 

Group 

Mean 

Standard  Deviation 

5 

Text 

213.60 

219.62 

No-Links 

311.78 

135.99 

Links 

567.00 

302.73 

(F  =  6.967.  n  =  36,  p  =  0.0030) 

There  were  a  total  of  38  males  and  22  females  participating  in  the  study.    Data  were  bro- 
ken down  by  gender,  but  there  were  no  significant  differences  between  males  and  females. 
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Attitudinal  and  experiential  measures  did  not  vary  according  to  whether  subjects  were  in 
the  No-Links  or  Links  groups.  Eight\-eight  percent  of  these  subjects  did  not  have  much  pre- 
vious experience  with  databases  and  could  thus  be  considered  comparable  on  this  dimension 
with  naive  or  occasional  users  in  other  studies.  Eighty  percent  of  these  subjects  had  also  a 
generally  favorable  attitude  towards  computers,  finding  them  at  least  somewhat  enjoyable. 
There  was  no  relationship  between  attitude  or  experience  and  performance. 

4.  Summary  and  Recommendations 

Some  questions  encouraged  the  use  of  links  more  than  did  others  and  some  subjects  in  the 
Links  group  were  more  likely  to  use  the  hypertext  menu  than  were  others.  This  finding  is 
consistent  with  that  of  Weyer  who  found  that  students  learned  simple  features  of  Dynabook 
quickly,  but  did  not  use  its  more  elaborate  features  [We82]. 

Subjects  in  the  No-Links  group  completed  significantly  more  queries  than  did  those  who 
used  links.  We  believe  that  this  is  not  merely  a  result  of  confusion  from  being  offered  the 
choice  of  several  alternatives,  because  subjects  who  had  the  option  to  use  hypertext  but  did 
not  do  so  performed  comparably  to  those  who  did  not  have  the  option  available. 

Comparisons  with  a  group  that  used  more  conventional  text  for  knowledge  exploration 
revealed  that  they  performed  best  of  all  groups  when  the  Table  of  Contents  was  enough  to 
locate  the  desired  information.  When  the  Table  of  Contents  was  not  enough,  the  Text  group 
performed  poorest  of  the  three  groups.  The  advantage  of  text  under  certain  circumstances  is 
consistent  with  research  reported  by  Shneiderman  [Shn87].  Reading  from  screens  has  been 
found  to  be  roughly  thirty  percent  slower  than  reading  from  paper  text. 

There  was  a  statistically  significant  positive  correlation  between  the  average  number  of 
specifications  used  for  a  query  and  the  average  number  of  seconds  spent  obtaining  the  answer 
to  a  query.  An  excessive  number  of  specifications  generally  indicated  an  inability  to  properly 
use  the  system  to  retrieve  the  desired  information. 

There  was  some  evidence  of  practice  effects  in  the  data  in  the  two  NetBook  groups 
through  the  fourth  query  completed.  Fatigue  may  have  increased  completion  time  for  the 
fifth  query.  There  was  no  evidence  of  practice  effects  in  the  Text  group,  but  none  was 
expected,  since  users  of  text  have  had  plenty  of  prior  practice  and  are  actually  expert  users  of 
the  medium,  rather  than  novices  as  was  the  case  with  both  electronic  groups. 

4.1.  Recommendations  on  the  Use  of  Links 

One  of  the  reasons  why  users  in  a  particular  system  have  difficulty  using  links  is  that  they 
do  not  always  have  all  of  the  information  that  they  need  or  not  all  of  it  is  salient.  As  Lan- 
dauer  et  al.  have  pointed  out,  users  may  receive  only  a  small  amount  of  local  detail  at  any 
one  time,  i.e.  what  a  fragment's  links  look  like,  whereas  more  global  information  on  the 
overall  structure  of  the  database  and  its  paths  of  links  might  be  more  useful  [LDGF82].  This 
is  consistent  with  subjects'  reactions  to  NetBook.  Some  subjects  expressed  the  fear  that  using 
links  would  get  them  too  far  afield.  They  felt  it  was  quicker  to  search  through  the  fragments 
called  up  by  properties  and  values  (which  are  global  information)  rather  than  to  follow  links 
through  "unknown  territory". 

Although  using  links  may  in  fact  take  longer  and  may  result  in  a  higher  number  of  naive 
users  getting  lost,  designers  of  information  systems  agree  that  it  is  highly  desirable  that  a  sys- 
tem have  this  feature,  particularly  when  properties  and  values  don't  fit  what  the  user  has  in 
mind. 
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Links,  however,  could  possibly  be  improved  in  one  of  the  following  ways: 

1)  A  facility  for  allowing  the  user  to  see  the  properties  and  values  of  a  specific  link  target 
could  be  helpful.  With  this  feature,  those  who  felt  reasonably  confident  about  their  original 
query  specifications  and  did  not  want  to  wander  off  into  a  different  topic  might  not  choose  a 
particularly  link.  On  the  other  hand,  users  who  felt  that  the  properties  and  values  of  the  link 
fragment  were  more  appropriate  than  their  original  specifications  could  then  modify  the 
direction  of  their  search.  In  the  present  study,  users  of  links  sometimes  wandered  out  of 
their  specification  set.  They  were  not  only  uncertain  about  whether  they  had  actually  left 
their  original  specification  set,  but  were  also  uncertain  about  how  to  get  back  to  their  original 
specifications  or  even  whether  it  was  desirable  to  get  back  to  the  original  specification  set. 
These  difficulties  could  be  alleviated  somewhat  with  the  proposed  facility. 

2)  A  "fisheye"  view  of  the  database,  as  proposed  by  Furnas,  could  be  presented  in  a  small 
window  on  the  user's  screen  [Fu83].  Users  would  have  a  simultaneous  view  of  both  the  parts 
of  the  database  structure  near  their  current  focus  of  interest  (e.g.  a  fragment's  links)  and  a 
view  of  the  higher  structure  of  the  database  (e.g.  major  attribute  headings  and  where  a  frag- 
ment specification  fits  into  that  overall  structure).  This  may  be  a  useful  way  to  keep  people 
oriented  in  an  information  system  like  NetBook  in  which  the  structure  is  too  large  for  naive 
users  to  deal  with  all  at  one  time.  In  the  present  study,  some  users  of  the  links  reported  feel- 
ing overwhelmed  by  the  attempt  to  keep  themselves  oriented  in  the  text,  often  just  giving  up 
and  randomly  exploring.  Orientation  aids  in  the  system  would  help  prevent  random  exploring 
in  situations  where  a  directed  search  to  find  specific  information  is  more  appropriate. 

3)  Embedding  links  directly  into  text,  as  Shneiderman  or  Brown  describe,  or  displaying 
link  names  at  the  bottom  of  the  screen,  vhile  a  text  fragment  is  displayed,  might  be  more 
efficient  means  for  entering  into  a  linked  oath  than  requesting  a  separate  hypertext  menu  as 
is  now  done  in  NetBook  [B86a,B86b,Shn8~1.  Some  users  in  the  present  study  seemed  to  have 
difficulty  moving  between  the  specification  menu  and  the  hypertext  menu.  Having  all  choices 
displayed  on  the  screen  at  once  would  eliminate  this  difficulty. 

In  addition,  the  way  the  data  is  organized  in  a  text  database  may  need  to  be  examined 
experimentally.  Categorization  of  attributes  and  values  of  text  segments  should  perhaps  ini- 
tially be  done  by  a  group  of  naive  users  similar  to  the  users  of  the  system.  The  optimum 
number  of  properties  and'or  values  should  also  be  examined  experimentally.  In  addition,  the 
optimum  amount  of  hierarchical  structure  could  be  explored.  This  may  vary  from  application 
to  application. 

Finally,  whether  or  not  links  are  desirable  may  depend  upon  how  a  system  is  to  be  used. 
In  information  systems,  undirected  explorations  without  an  answer  to  a  specific  question  in 
mind  are  different  from  directed  searches  for  a  specific  piece  of  information.  Links  may  be 
more  useful  when  searches  are  undirected.  A  person  who  is  generally  interested  in  a  topic 
might  follow  the  links  that  pique  his  curiosity,  without  worrying  too  much  about  getting  lost. 
But  faced  with  a  specific  question  and  a  set  of  properties  and  values  to  which  the  query  maps 
reasonably  well,  the  user  may  be  better  off  not  using  the  links. 

5.  Acknowledgement 

The  NetBook  system  was  implemented  by  Leah  Beckman,  Jose  Perez-Carballo  and  Dana 
Birkby.    We  wish  to  thank  them  for  all  their  help. 


16 


REFERENCES 

[AM84]  R.  M.  Akscyn  and  D.  L.  McCracken,  "The  ZOG  approach  to  database  manage- 
ment," In  Proceedings  of  the  Trends  and  Applications  Conference:  Making  Database  Work, 
Maryland,  1984. 

[B86a]  P.  J.  Brown,  "Interactive  documentation,"  Software  -  Practice  and  Experience,  16, 
1986,  291-299. 

[B86b]  P.  J.  Brown,  "A  simple  mechanism  for  authorship  of  dynamic  documents, "  In  Text 
Processing  and  Document  .Manipulation,  Proceedings  of  the  International  Conference,  ed.  J. 
C.  van  Vliet,  1986. 

[Bu45]  V.  Bush,  "As  we  may  think."  Atlantic  Monthly,  July,  1945. 

[CSC87]  A.  J.  Canas,  F.  R.  Safayeni,  and  D.  W.  Conrath,  "A  conceptual  model  and 
experiments  on  how  people  classify  and  retrieve  documents,"  Paper  presented  at  the  8th 
Annual  International  ACM  SIGIR  Conference,  Montreal,  1985. 

[CMN83]  S.  K.  Card,  T.  P.  .Moran,  and  A.  Newell,  The  Psychology  of  Human-Computer 
Interaction,  Lawrence  Erlbaum  Associates,  1983. 

[C80]  R.  G.  G.  Cattell,  ".An  entity-based  database  user  interface,"  In  Proceedings  of  the 
ACM  SIGMOD  Conference  on  Management  of  Data.  1980. 

[CAL83]  A.  Chapanis,  N.  S.  Anderson,  and  J.  C.  R.  Licklider,  "User-computer  interac- 
tion. Research  Needs  for  Human  Factors,  Committee  on  Human  Factors,"'  National  Research 
Council,  National  Academy  Press,  1983. 

[DL84]  S.  T.  Dumais  and  T.  K.  Landauer,  ""Describing  categories  of  objects  for  menu 
retrieval  systems,"  Behavioral  Research  Methods,  Instruments  and  Computers,  16,  1984, 
242-248. 

[E84]  D.  C.  Engelbart,  "Collaboration  support  provisions  in  AUGMENT,"  In  AFIPS 
Office  Automation  Conference  Digest,  1984,  51-58. 

[F84]  D.  Fogg,  "Lessons  from  a  'Living  in  a  Database'  graphical  query  interface,'"  In 
Proceedings  of  the  ACM  SIGMOD  Conference  on  Management  of  Data,  1984. 

[FV80]  J.  L.  Frankhuizen  and  N.  Vrins,  "Human  factors  studies  with  Viewdata,"  In 
Proceedings  of  the  Ninth  International  Symposium  on  Human  Factors  in  Telecommunica- 
tions, 1980,  pp. 1-7. 

[Fu83]  G.  W.  Furnas,  "Small  Views  of  Large  Structures,"  In  Workshop  of  Intelligent 
User  Interfaces,  New  Hampshire,  1983. 

[H80]  C.  F.  Herot,  "Spatial  management  of  data,"  ACM  Transactions  on  Database  Sys- 
tems, 5,  1980,  493-513. 

[LDGF82]  T.  K.  Landauer,  S.  T.  Dumais,  L.  M.  Gomez,  and  G.  W.  Furnas,  '"Human 
factors  in  data  access, "  The  Bell  System  Technical  Journal,  61,  (1982),  2487-2509. 

[L85]   D     G.    Lowe,    "Co-operative    structuring   of   information:     the    representation   of 


17 


reasoning  and  debate, '  International  Journal  of  Man-Machine  Studies,  23.  1985,  97-111. 

[M87]  M.  T.  Motley,  "What  I  meant  to  say,"  Psychology  Today,  February  (1987). 

[N79]  N.  Negroponte,  'Books  without  pages,"  ICC  International  Conference  on  Com- 
munications, IV,  1979  56.1.1-56.1.8. 

[Ne841  T.  Nelson,  Literary  .Machines.  T.  Nelson,  P.O.  Box  118,  Swarthmore,  PA,  19081, 
1984. 

[No83]  D.  A.  Norman.  "Design  principles  for  human-computer  interaction,"  In  ACM 
CHr83  Proceedings,  Human  Factors  in  Computing  Systems,  1983. 

fS85]  D.  Shasha,  "NetBook  -  a  data  model  to  support  knowledge  exploration,"  In 
Proceedings  of  the  11th  International  Conference  on  Very  Large  Data  Bases,  eds.  A.  Pirotte 
and  Y.  Vassilious,  1985. 

[S86|  D.  Shasha,  "When  does  non-linear  text  help?"  In  Proceedings  of  the  First  Interna- 
tional Conference  on  Expert  Database  Systems,  ed.  L.  Kirschberg,  1986. 

[Shn80]  B.  Shneiderman,  Software  Paychology,  Little,  Brown  and  Company,  1980. 

[Shn87]  B.  Shneiderman,  "User  Interface  Design  and  Evaluation  for  an  Electronic  Ency- 
clopedia," In  Second  International  Conference  on  Human-Computer  Interaction,  North- 
Holland,  1987. 

[TW86]  R.  H.  Trigg  and  .M.  Weiser,  "TEXTNET:  A  network-based  approach  to  text 
handling,"    ACM  Transactions  on  Office  Information  Systems,  4,  1986,  1-23. 

[W81]  D.  E.  Walker,  "The  organization  and  use  of  information;  contributions  of  informa- 
tion science,  computational  linguistics  and  artificial  intelligence,"  Journal  of  the  American 
Society  for  Information  Science,  32,  1981  347-363. 

[We82]  S.  A.  Weyer,  "The  design  of  a  dynamic  book  for  information  search,"  Interna- 
tional Journal  of  Man-.Machine  Studies,  17,  1982,  87-107. 

[Y.M851  N.  Yankelovich  and  N.  Meyrowitz,  "Reading  and  writing  the  electronic  book," 
IEEE  Computer,  18,  1985,  15-32. 


18 


^U  COMPsci   TR-309 
Gray,   susan  H       °'  =•! 

°  ''"'  °'  not   to  u.,. 


^°   """or„o\    ,„   ^^   ^ 


■^'"•^  bnolc  m„y  1,^  ^^pj 

FOURTEEN    DAYc: 


'f""- each  d.iy  the  hook 


«  '<fPt  overtime. 


/-vsV.^';: 


Mtmm 


m^>n 


i  W  ;T.fl'ZT  WZsmW/  t*^  ' 


